Augmented three dimensional point collection of vertical structures

ABSTRACT

Automated methods and systems are disclosed, including a method comprising: capturing images and three-dimensional LIDAR data of a geographic area with an image capturing device and a LIDAR system, the images depicting an object of interest and the three-dimensional LIDAR data including the object of interest, the image capturing device capturing the images of a vertical surface of the object of interest at one or more oblique angle, and the LIDAR system capturing the three-dimensional LIDAR data of a horizontal surface of the object of interest at a nadir angle; analyzing the images with a computer system to determine three dimensional locations of points on the object of interest; and updating the three-dimensional LIDAR data with the three dimensional locations of points on the object of interest determined by analyzing the images to create a 3D point cloud having a resolution greater than a resolution of the three-dimensional LIDAR data.

INCORPORATION BY REFERENCE

The present patent application claims priority to and is a continuationof U.S. Ser. No. 15/364,812, filed on Nov. 30, 2016; which is acontinuation of U.S. Ser. No. 15/060,264, filed on Mar. 3, 2016, whichissued as U.S. Pat. No. 9,542,738 on Jan. 10, 2017; which is adivisional of the patent application identified by U.S. Ser. No.14/169,872, filed Jan. 31, 2014, which issued as U.S. Pat. No. 9,292,913on Mar. 22, 2016; the entire contents of all of which are herebyincorporated herein by reference.

BACKGROUND

The utility industry continually tracks and measures physical assets ofits networks (e.g., utility wires, utility poles, utility towers), andassesses the current conditions of those assets. With tracking andmeasurement, the industry seeks to understand information on the currentstate of the utilities including infringement rights, growth ofvegetation, and the like.

Currently, assessment of the utility corridor includes the use of groundcrews that walk or drive along the right of way. Companies may also useanything from helicopter flights carrying experts observing assets fromthe air, to aerial sensor platforms capturing photographic, positional,or other information through the use of remote sensing technology.

Remote sensing technology may have the ability to be the most costeffective while providing pertinent information for assessment of theutility corridor. Cost efficiency may be increased further with captureefficiency. For example, cost efficiency may be increased by usingfaster aircraft (e.g., fixed wing aircraft), allowing for collection ofdata over a large number of utility line miles, and the like.Additionally, the use of multiple sensors may aid in collecting largeamounts of sensor data, such as, for example, visible cameras, infra-redcameras, and LIDAR scanners.

One direction that the utility industry is developing is modeling assetsand features in three dimensions. One base representation of thisstructure is known as a Method 1 structure model. Currently, this isproduced by collecting three-dimensional data points through the use ofa LIDAR scanner. By flying low and slow, helicopter systems capture 10to 20 points per square meter, producing dense point grids. Even at 40points per grid, however, the average spacing between each point may be15-cm or about 6 inches. For smaller structures, this may causemeasurement inaccuracy.

While lasers have been achieving higher pulse frequencies, there arephysical limitations to collecting higher and denser three-dimensionalpoint clouds from a LIDAR scanner. First, the high density point cloudsmay require flying lower and slower, running counter to a goal of higherefficiency. Second, in order to achieve the higher pulse repetitionrates, multiple pulses may need to be in the air simultaneously. Eventhough light travels extremely quickly, it may take a set time to reachthe ground and reflect back to the sensor of the LIDAR scanner. If toomany pulses are in the air simultaneously, subsequent pulses may causeinterference.

Traditional LIDAR scanner collection methods typically direct and orientthe LIDAR collection system straight down (i.e., nadir). This may onlyallow for 10 to 20 points per square meter on the ground or on ahorizontal structure. When vertical structures are present, however, thepoint density is even further reduced. For a fully vertical surface, theLIDAR scanner may only collect points prior to the vertical structureand on a horizontal surface of the structure at the vertical top. Toproduce vertical points, the LIDAR scanner may be tilted at an angle,however, now either multiple LIDAR system may need to be installed tocapture multiple sides of the structure, or a conical collection pathmay need to be collected as described in a patent application identifiedby U.S. Ser. No. 13/797,172 that was filed on Mar. 12, 2013, and issuedas U.S. Pat. No. 9,244,272 on Jan. 26, 2016, which is herebyincorporated by reference in its entirety.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

To assist those of ordinary skill in the relevant art in making andusing the subject matter hereof, reference is made to the appendeddrawings, which are not intended to be drawn to scale, and in which likereference numerals are intended to refer to similar elements forconsistency. For purposes of clarity, not every component may be labeledin every drawing.

FIG. 1 illustrates an exemplary embodiment of a platform or vehiclecarrying an image-capturing system and illustrates exemplary orthogonaland oblique images taken thereby.

FIG. 2 is a diagrammatic view of the image-capturing system of FIG. 1.

FIG. 3 is a block diagram of the image-capturing computer system of FIG.2.

FIG. 4 is a block diagram of the image-capturing computer system of FIG.2 communicating via a network with multiple processors and ageographical information system (GIS) data system.

FIG. 5 is an exemplary LIDAR 3D point cloud depiction illustratingclassification of structures therein.

FIG. 6 is an exemplary diagram illustrating an exemplary utility towerhaving utility wires, a cross bar, and insulators.

FIG. 7 is another exemplary LIDAR 3D point cloud depiction illustratingparabolas fitted to adjacent utility wires, wherein the location ofintersection of the parabolas estimates the location of a utility tower.

FIG. 8A and FIG. 8B are exemplary LIDAR 3D point clouds illustratinglocation and identification of clusters as utility wires and/or crossbars within the utility corridor.

FIG. 9 is a side view of the utility tower illustrated in FIG. 6 havinga TGP vertical plane provided therethrough.

FIG. 10 is an exemplary diagrammatic view illustrating multiple raysprojected from a platform to objects of interest on a utility towerbased on the view of an oblique image, the rays intersecting the TGPvertical plane of the utility tower.

FIG. 11A is another exemplary diagrammatic view illustrating a singleray projected from a platform to an object of interest on a utility polebased on the view of an oblique image, the ray intersecting the TGPvertical plane of the utility tower.

FIG. 11B is a diagrammatic view illustrating boundaries of the opposingview of the oblique image illustrated in FIG. 11A.

FIG. 12 is an exemplary nadir image illustrating utility wires and across bar.

FIG. 13A is an exemplary image produced after a Gabor Filter is appliedto the utility wires in the nadir image of FIG. 12.

FIG. 13B is an exemplary image produced after a maximum responsethreshold is applied to the image of FIG. 13A providing detected utilitywires.

FIG. 14A is an exemplary image produced after a Gabor Filter is appliedto the cross bar in the nadir image of FIG. 12.

FIG. 14B is an exemplary image produced after a maximum responsethreshold is applied to the image of FIG. 14A providing a detected crossbar.

FIG. 15A is an exemplary image produced after overlapping the images ofFIG. 13B and FIG. 14B illustrating detected utility wires and a detectedcross bar.

FIG. 15B is an exemplary image of the detected utility wires anddetected cross bar of FIG. 15A having an extension applied to thedetected cross bar.

FIG. 16 is an exemplary oblique image having the detected cross bar ofFIG. 14B positioned therein.

FIG. 17 is another exemplary oblique image having the detected cross barof FIG. 14B positioned therein, the oblique images of FIG. 16 and FIG.17 being opposing views.

FIGS. 18A-18D illustrate an exemplary image displayed on the system ofFIG. 2, and the use of an exemplary template for aligning to a utilitytower within the image.

FIG. 19 is a diagrammatic view illustrating boundaries of two successiveoblique images for finding additional three-dimensional points on thesurface of a utility tower.

FIG. 20 is a LIDAR 3D point cloud generated from stereo pair obliqueimages showing ground points and utility tower points.

DETAILED DESCRIPTION

Before explaining at least one embodiment of the disclosure in detail,it is to be understood that the disclosure is not limited in itsapplication to the details of construction, experiments, exemplary data,and/or the arrangement of the components set forth in the followingdescription or illustrated in the drawings unless otherwise noted.

The disclosure is capable of other embodiments or of being practiced orcarried out in various ways. Also, it is to be understood that thephraseology and terminology employed herein is for purposes ofdescription, and should not be regarded as limiting.

The following detailed description refers to the accompanying drawings.The same reference numbers in different drawings may identify the sameor similar elements.

As used in the description herein, the terms “comprises,” “comprising,”“includes,” “including,” “has,” “having,” or any other variationsthereof, are intended to cover a non-exclusive inclusion. For example,unless otherwise noted, a process, method, article, or apparatus thatcomprises a list of elements is not necessarily limited to only thoseelements, but may also include other elements not expressly listed orinherent to such process, method, article, or apparatus.

As used in the instant disclosure, the terms “provide”, “providing”, andvariations thereof comprise displaying or providing for display awebpage (e.g., webpage having one or more images and software to permitmeasurement within the images), electronic communications, e-mail,and/or electronic correspondence to one or more user terminalsinterfacing with a computer and/or computer network(s) and/or allowingthe one or more user terminal(s) to participate, such as by interactingwith one or more mechanisms on a webpage, electronic communications,e-mail, and/or electronic correspondence by sending and/or receivingsignals (e.g., digital, optical, and/or the like) via a computer networkinterface (e.g., Ethernet port, TCP/IP port, optical port, cable modem,combinations thereof, and/or the like). A user may be provided with aweb page in a web browser, or in a software application, for example.

Further, unless expressly stated to the contrary, “or” refers to aninclusive and not to an exclusive “or”. For example, a condition A or Bis satisfied by one of the following: A is true (or present) and B isfalse (or not present), A is false (or not present) and B is true (orpresent), and both A and B are true (or present).

In addition, use of the “a” or “an” are employed to describe elementsand components of the embodiments herein. This is done merely forconvenience and to give a general sense of the inventive concept. Thisdescription should be read to include one or more, and the singular alsoincludes the plural unless it is obvious that it is meant otherwise.Further, use of the term “plurality” is meant to convey “more than one”unless expressly stated to the contrary.

As used herein, any reference to “one embodiment,” “an embodiment,”“some embodiments,” “one example,” “for example,” or “an example” meansthat a particular element, feature, structure or characteristicdescribed in connection with the embodiment is included in at least oneembodiment. The appearance of the phrase “in some embodiments” or “oneexample” in various places in the specification is not necessarily allreferring to the same embodiment, for example.

Circuitry, as used herein, may be analog and/or digital components, orone or more suitably programmed processors (e.g., microprocessors) andassociated hardware and software, or hardwired logic. Also, “components”may perform one or more functions. The term “component,” may includehardware, such as a processor (e.g., microprocessor), an applicationspecific integrated circuit (ASIC), field programmable gate array(FPGA), a combination of hardware and software, and/or the like.

Software may include one or more computer readable instructions thatwhen executed by one or more components cause the component to perform aspecified function. It should be understood that the algorithmsdescribed herein may be stored on one or more non-transient memory.Exemplary non-transient memory may include random access memory, readonly memory, flash memory, and/or the like. Such non-transient memorymay be electrically based, optically based, and/or the like.

It is to be further understood that, as used herein, the term user isnot limited to a human being, and may comprise, a computer, a server, awebsite, a processor, a network interface, a human, a user terminal, avirtual computer, combinations thereof, and the like, for example.

Referring now to the Figures, and in particular to FIG. 1, shown thereinis a schematic diagram of hardware forming an exemplary embodiment of anapparatus 10 for three-dimensional point collection of verticalstructures. The apparatus 10 may include a platform and/or vehicle 12carrying an image-capturing and geo-locating system 14.

The platform 12 may be an airplane, space shuttle, rocket, satellite, orany other suitable vehicle capable of carry the image-capturing system14. For example, in some embodiments, the platform 12 may be a fixedwing aircraft.

The platform 12 may carry the image-capturing system 14 over an area ofand at one or more altitudes above a surface 16. For example, theplatform 12 may carry the image-capturing system 14 over a predefinedarea and at one or more predefined altitudes above the Earth's surfaceand/or any other surface of interest.

The platform 12 may be capable of controlled movement and/or flight. Assuch, the platform 12 may be manned or unmanned. In some embodiments,the platform 12 may be capable of controlled movement and/or flightalong a pre-defined flight path and/or course. For example, the platform12 may be capable of controlled movement and/or flight along the Earth'satmosphere and/or outer space. In some embodiments, the platform 12 maybe capable of controlled movement and/or flight along a utilitycorridor.

The platform 12 may include a system for generating and/or regulatingpower. For example, the platform 12 may include one or more generators,fuel cells, solar panels, and/or batteries for powering theimage-capturing and geo-locating system 14.

Referring to FIGS. 1 and 2, the image-capturing and geo-locating system14 may include two or more oblique image capturing devices 18 a and 18b, one or more vertical image-capturing devices 20, one or more LIDARscanners 22, one or more global positioning system (GPS) receivers 24,one or more inertial navigation units (INU) 26, one or more clocks 28,one or more gyroscopes 30, one or more compasses 32, one or morealtimeters 34. In some embodiments, each of the elements of theimage-capturing and geo-locating system 14 may be interconnected with animage-capturing computer system 36.

Generally, the oblique image-capturing devices 18 a and 18 b and thevertical image-capturing device 20 may be capable of capturing imagesphotographically and/or electronically. The oblique image-capturingdevices 18 a and 18 b and the vertical image-capturing device 20 mayinclude, but are not limited to, conventional cameras, digital cameras,digital sensors, charge-coupled devices, and/or the like. In someembodiments, the oblique image-capturing devices 18 a and 18 b and thevertical image-capturing device 20 may be an ultra-high resolutioncameras. For example, in some embodiments, the oblique image-capturingdevices 18 a and 18 b may be ultra-high resolution oblique capturesystems, such as may be found in the Pictometry PentaView CaptureSystem, manufactured and distributed by Pictometry International basedin Henrietta, N.Y. Similarly, in some embodiments, the verticalimage-capturing device 20 may also be a high resolution vertical capturesystem, such as may be found in the Pictometry PentaView Capture System.

The oblique image-capturing devices 18 a and 18 b and the verticalimage-capturing device 20 may include known or determinablecharacteristics including, but not limited to, focal length, sensorsize, aspect ratio, radial and other distortion terms, principal pointoffset, pixel pitch, alignment, and/or the like.

The oblique image-capturing devices 18 a and 18 b may include respectivecentral axes A₁ and A₂. In some embodiments, the oblique image-capturingdevices 18 a and 18 b may be mounted to the platform 12 such that axesA₁ and A₂ each may be at an angle of declination Θ relative to ahorizontal plane P as illustrated in FIG. 1. Declination angle Θ may beany oblique angle. Generally, declination angle Θ may be fromapproximately 20° (twenty degrees) to approximately 60° (sixty degrees).In some embodiments, the declination angle Θ may be approximately 45°(forty-five degrees).

The vertical image-capturing device 20 may include central axis A₃. Insome embodiments, the vertical image-capturing device 20 may be mountedto the platform 12 such that the angle of declination Θ relative to ahorizontal plane P of axis A₃ is approximately 90° (ninety degrees). Assuch, the vertical image-capturing device 20 may generally be mounted atnadir.

The oblique image-capturing devices 18 a and 18 b may acquire one ormore oblique images and issue one or more image data signals (IDS) 40 aand 40 b corresponding to one or more particular oblique images oroblique photographs taken. The vertical image-capturing device 20 mayacquire one or more nadir images and issue one or more image datasignals (IDS) 42 corresponding to one or more particular nadir images ornadir photographs taken. Oblique images and/or nadir images may bestored in the image-capturing computer system 36.

The LIDAR scanner 22 may determine a distance between the platform 12and an object of interest by illuminating the object of interest with alaser and analyzing the reflected light. An exemplary LIDAR scanner 22may be the Riegl LMS-Q680i, manufactured and distributed by Riegl LaserMeasurement Systems located in Horn, Austria. In some embodiments, theLIDAR scanner 22 may be a downward projecting high pulse rate LIDARscanning system.

In some embodiments, the LIDAR scanner 22 may be mounted in anoff-vertical position on the platform 12. For example, the LIDAR scanner22 may be mounted to the platform 12 such that axis A₄ may be at anangle of declination Θ relative to a horizontal plane P. Declinationangle Θ may be any oblique angle. In some embodiments, the declinationangle Θ may be any angle less than or equal to 80 degrees such that theaxis A₄ is roughly 10 degrees or more up from nadir in either a forwardor rearward direction. Mounting in an off-vertical position (i.e.,non-nadir) may aid in obtaining points on a face of a vertical structureas described in further detail herein. In some embodiments, the LIDARscanner 22 may collect on average between 5 and 10 points per squaremeter.

Alternatively, a helical scan LIDAR system may be used in lieu of, or inaddition to, the LIDAR scanner 22. The helical scan LIDAR system may bemounted such that at least one portion of the scan pattern may beroughly 10 degrees or more up from nadir.

The GPS receiver 24 may receive global positioning system (GPS) signals48 that may be transmitted by one or more global positioning systemsatellites 50. The GPS signals 48 may enable the location of theplatform 12 relative to the surface 16 and/or an object of interest tobe determined. The GPS receiver 24 may decode the GPS signals 48 and/orissue location signals and/or data 52. The location signals and/or data52 may be dependent, at least in part, on the GPS signals 48 and may beindicative of the location of the platform 12 relative to the surface 16and/or an object of interest. The location signals and/or data 52corresponding to each image captured by the oblique image-capturingdevices 18 a and 18 b and/or the vertical image-capturing device 20 maybe received and/or stored by the image-capturing computer system 36 in amanner in which the location signals are associated with thecorresponding image.

The INU 26 may be a conventional inertial navigation unit. The INU 26may be coupled to and detect changes in the velocity (e.g.,translational velocity, rotational velocity) of the oblique imagecapturing devices 18 a and 18 b, the vertical image-capturing devices20, the LIDAR scanner 22, and/or the platform 12. The INU 26 may issuevelocity signals and/or data 54 indicative of such velocities and/orchanges therein to image-capturing computer system 36. Theimage-capturing computer system 36 may then store the velocity signalsand/or data 54 corresponding to each oblique and/or nadir image capturedby the oblique image-capturing devices 18 a and 18 b, the verticalimage-capturing device 20, and/or points collected by the LIDAR scanner22.

The clock 28 may keep a precise time measurement. For example, the clock28 may keep a precise time measurement used to synchronize events withinthe image capturing and geo-locating system 14. The clock 28 may includea time data/clock signal 56. In some embodiments, the time data/clocksignal 56 may include a precise time that an oblique and/or nadir imageis taken by the oblique image-capturing devices 18 a and 18 b and/or thevertical image-capturing device 20, and/or the precise time that pointsare collected by the LIDAR scanner 22. The time data 56 may be receivedby and/or stored by the image-capturing computer system 36. In someembodiments, the clock 28 may be integral with the image-capturingcomputer system 36, such as, for example, a clock software program.

The gyroscope 30 may be a conventional gyroscope commonly found onairplanes and/or within navigation systems (e.g., commercial navigationsystems for airplanes). Gyroscope 30 may submit signals including a yawsignal 58, a roll signal 60, and/or a pitch signal 62. In someembodiments, the yaw signal 58, the roll signal 60, and/or the pitchsignal 62 may be indicative of the yaw, roll and pitch of the platform12. The yaw signal 58, the roll signal 60, and/or the pitch signal 62may be received and/or stored by the image-capturing computer system 36.

The compass 32 may be any conventional compass (e.g., conventionalelectronic compass) capable of indicating the heading of the platform12. The compass 32 may issue a heading signal and/or data 64. Theheading signal and/or data 64 may be indicative of the heading of theplatform 12. The image-capturing computer system 36 may receive, storeand/or provide the heading signal and/or data 64 corresponding to eachoblique and/or nadir image captured by the oblique image-capturingdevices 18 a and 18 b and/or the vertical image-capturing device 20.

The altimeter 34 may indicate the altitude of the platform 12. Thealtimeter 34 may issue an altimeter signal and/or data 66. Theimage-capturing computer system 36 may receive, store and/or provide thealtimeter signal and/or data 66 corresponding to each oblique and/ornadir image captured by the oblique image-capturing devices 18 a and 18b, and/or the vertical image-capturing device 20.

Referring to FIGS. 2 and 3, the image-capturing computer system 36 maybe a system or systems that are able to embody and/or execute the logicof the processes described herein. Logic embodied in the form ofsoftware instructions and/or firmware may be executed on any appropriatehardware. For example, logic embodied in the form of softwareinstructions or firmware may be executed on a dedicated system orsystems, or on a personal computer system, or on a distributedprocessing computer system, and/or the like. In some embodiments, logicmay be implemented in a stand-alone environment operating on a singlecomputer system and/or logic may be implemented in a networkedenvironment, such as a distributed system using multiple computersand/or processors.

In some embodiments, the image-capturing computer system 36 may includeone or more processors 70 communicating with one or more image capturinginput devices 72, image capturing output devices 74, and/or I/O ports 76enabling the input and/or output of data to and from the image-capturingcomputer system 36.

FIG. 3 illustrates the image-capturing computer system 36 having asingle processor 70. It should be noted, however, that theimage-capturing computer system 36 may include multiple processors 70.In some embodiments, the processor 70 may be partially or completelynetwork-based or cloud-based. The processor 70 may or may not be locatedin a single physical location. Additionally, multiple processors 70 mayor may not necessarily be located in a single physical location.

The one or more image capturing input devices 72 may be capable ofreceiving information input from a user and/or processor(s), andtransmitting such information to the processor 70. The one or more imagecapturing input devices 72 may include, but are not limited to,implementation as a keyboard, touchscreen, mouse, trackball, microphone,fingerprint reader, infrared port, slide-out keyboard, flip-outkeyboard, cell phone, PDA, video game controller, remote control, faxmachine, network interface, speech recognition, gesture recognition, eyetracking, brain-computer interface, combinations thereof, and/or thelike.

The one or more image capturing output devices 74 may be capable ofoutputting information in a form perceivable by a user and/orprocessor(s). For example, the one or more image capturing outputdevices 74 may include, but are not limited to, implementations as acomputer monitor, a screen, a touchscreen, a speaker, a website, atelevision set, a smart phone, a PDA, a cell phone, a fax machine, aprinter, a laptop computer, an optical head-mounted display (OHMD),combinations thereof, and/or the like. It is to be understood that insome exemplary embodiments, the one or more image capturing inputdevices 72 and the one or more image capturing output devices 74 may beimplemented as a single device, such as, for example, a touchscreen or atablet.

Each of the data signals 40 a, 40 b, 42, 46, 52, 54, 56, 58, 60, 62,and/or 64 may be provided to the image capturing computer system 36. Forexample, each of the data signals 40 a, 40 b, 42, 46, 52, 54, 56, 58,60, 62, and/or 64 may be received by the image capturing computer system36 via the I/O port 76. The I/O port may comprise one or more physicaland/or virtual ports.

In some embodiments, the image-capturing computer system 36 may be incommunication with one or more additional processors 82 as illustratedin FIG. 4. In this example, the image-capturing computer system 36 maycommunicate with the one or more additional processors 82 via a network80. As used herein, the terms “network-based”, “cloud-based”, and anyvariations thereof, may include the provision of configurablecomputational resources on demand via interfacing with a computer and/orcomputer network, with software and/or data at least partially locatedon the computer and/or computer network, by pooling processing power oftwo or more networked processors.

In some embodiments, the network 80 may be the Internet and/or othernetwork. For example, if the network 80 is the Internet, a primary userinterface of the image capturing software and/or image manipulationsoftware may be delivered through a series of web pages. It should benoted that the primary user interface of the image capturing softwareand/or image manipulation software may be replaced by another type ofinterface, such as, for example, a Windows-based application.

The network 80 may be almost any type of network. For example, thenetwork 80 may interface by optical and/or electronic interfaces, and/ormay use a plurality of network topographies and/or protocols including,but not limited to, Ethernet, TCP/IP, circuit switched paths, and/orcombinations thereof. For example, in some embodiments, the network 80may be implemented as the World Wide Web (or Internet), a local areanetwork (LAN), a wide area network (WAN), a metropolitan network, awireless network, a cellular network, a Global System for MobileCommunications (GSM) network, a code division multiple access (CDMA)network, a 3G network, a 4G network, a satellite network, a radionetwork, an optical network, a cable network, a public switchedtelephone network, an Ethernet network, combinations thereof, and/or thelike. Additionally, the network 80 may use a variety of networkprotocols to permit bi-directional interface and/or communication ofdata and/or information. It is conceivable that in the near future,embodiments of the present disclosure may use more advanced networkingtopologies.

The image capturing computer system 36 may be capable of interfacingand/or communicating with the one or more computer systems includingprocessors 82 via the network 80. Additionally, the one or moreprocessors 82 may be capable of communicating with each other via thenetwork 80. For example, the image capturing computer system 36 may becapable of interfacing by exchanging signals (e.g., analog, digital,optical, and/or the like) via one or more ports (e.g., physical ports orvirtual ports) using a network protocol, for example.

The processors 82 may include, but are not limited to implementation asa variety of different types of computer systems, such as a serversystem having multiple servers in a configuration suitable to provide acommercial computer based business system (such as a commercialweb-site), a personal computer, a smart phone, a network-capabletelevision set, a television set-top box, a tablet, an e-book reader, alaptop computer, a desktop computer, a network-capable handheld device,a video game console, a server, a digital video recorder, a DVD player,a Blu-Ray player, a wearable computer, a ubiquitous computer,combinations thereof, and/or the like. In some embodiments, the computersystems comprising the processors 82 may include one or more inputdevices 84, one or more output devices 86, processor executable code,and/or a web browser capable of accessing a website and/or communicatinginformation and/or data over a network, such as network 80. The computersystems comprising the one or more processors 82 may include one or morenon-transient memory comprising processor executable code and/orsoftware applications, for example. The image capturing computer system36 may be modified to communicate with any of these processors 82 and/orfuture developed devices capable of communicating with the imagecapturing computer system 36 via the network 80.

The one or more input devices 84 may be capable of receiving informationinput from a user, processors, and/or environment, and transmit suchinformation to the processor 82 and/or the network 80. The one or moreinput devices 84 may include, but are not limited to, implementation asa keyboard, touchscreen, mouse, trackball, microphone, fingerprintreader, infrared port, slide-out keyboard, flip-out keyboard, cellphone, PDA, video game controller, remote control, fax machine, networkinterface, speech recognition, gesture recognition, eye tracking,brain-computer interface, combinations thereof, and/or the like.

The one or more output devices 86 may be capable of outputtinginformation in a form perceivable by a user and/or processor(s). Forexample, the one or more output devices 86 may include, but are notlimited to, implementations as a computer monitor, a screen, atouchscreen, a speaker, a website, a television set, a smart phone, aPDA, a cell phone, a fax machine, a printer, a laptop computer, anoptical head-mounted display (OHMD), combinations thereof, and/or thelike. It is to be understood that in some exemplary embodiments, the oneor more input devices 84 and the one or more output devices 86 may beimplemented as a single device, such as, for example, a touchscreen or atablet.

Referring to FIGS. 2 and 3, in some embodiments, the image-capturingcomputer system 36 may include one or more processors 70 workingtogether, or independently to execute processor executable code, and oneor more memories 90 capable of storing processor executable code. Insome embodiments, each element of the image-capturing computer system 36may be partially or completely network-based or cloud-based, and may ormay not be located in a single physical location.

The one or more processors 70 may be implemented as a single orplurality of processors working together, or independently, to executethe logic as described herein. Exemplary embodiments of the one or moreprocessors 70 may include, but are not limited to, a digital signalprocessor (DSP), a central processing unit (CPU), a field programmablegate array (FPGA), a microprocessor, a multi-core processor, and/orcombination thereof, for example. The one or more processors 70 may becapable of communicating via the network 80, illustrated in FIG. 4, byexchanging signals (e.g., analog, digital, optical, and/or the like) viaone or more ports (e.g., physical or virtual ports) using a networkprotocol. It is to be understood, that in certain embodiments, usingmore than one processor 70, the processors 70 may be located remotelyfrom one another, in the same location, or comprising a unitarymulti-core processor. The one or more processors 70 may be capable ofreading and/or executing processor executable code and/or capable ofcreating, manipulating, retrieving, altering, and/or storing datastructures into one or more memories 90.

The one or more memories 90 may be capable of storing processorexecutable code. Additionally, the one or more memories 90 may beimplemented as a conventional non-transient memory, such as, forexample, random access memory (RAM), a CD-ROM, a hard drive, a solidstate drive, a flash drive, a memory card, a DVD-ROM, a floppy disk, anoptical drive, combinations thereof, and/or the like, for example.

In some embodiments, the one or more memories 90 may be located in thesame physical location as the image capturing computer system 36.Alternatively, one or more memories 90 may be located in a differentphysical location as the image capturing computer system 36, the withimage capturing computer system 36 communicating with one or morememories 90 via a network such as the network 80, for example.Additionally, one or more of the memories 90 may be implemented as a“cloud memory” (i.e., one or more memories 90 may be partially orcompletely based on or accessed using a network, such as network 80, forexample).

Referring to FIGS. 2 and 3, the one or more memories 90 may storeprocessor executable code and/or information comprising one or moredatabases 92 and program logic 94. In some embodiments, the processorexecutable code may be stored as a data structure, such as a databaseand/or data table, for example.

In use, the image-capturing computer system 36 may execute the programlogic 94 which may control the reading, manipulation, and/or storing ofdata signals 40 a, 40 b, 42, 46, 52, 54, 56, 58, 60, 62, and/or 64. Forexample, the program logic may read data signals 40 a, 40 b, and/or 42,and may store them within the one or more memories 90. Each of thelocation signals, 46, 52, 54, 56, 58, 60, 62, and/or 64, may representthe conditions existing at the instance that an oblique image and/ornadir image is acquired and/or captured by the oblique image capturingdevices 18 a and/or 18 b, and/or the vertical image-capturing device 20.

In some embodiments, the image capturing computer system 36 may issue animage capturing signal to the oblique image-capturing devices 18 aand/or 18 b, and/or the vertical image-capturing device 20 to therebycause those devices to acquire and/or capture an oblique image and/or anadir image at a predetermined location and/or at a predeterminedinterval. In some embodiments, the image capturing computer system 36may issue the image capturing signal dependent on at least in part onthe velocity of the platform 12. Additionally, the image capturingcomputer system 36 may issue a point collection signal to the LIDARscanner 22 to thereby cause the LIDAR scanner to collect points at apredetermined location and/or at a predetermined interval.

Program logic 94 of the image capturing computer system 36 may decode,as necessary, and/or store the aforementioned signals within the memory90, and/or associate the data signals with the corresponding image datasignals 40 a, 40 b and/or 42, or the corresponding LIDAR scanner signals46. Thus, for example, the altitude, orientation, roll, pitch, yaw, andthe location of each oblique image capturing device 18 a and 18 b,and/or vertical image-capturing device 20 relative to the surface 16and/or object of interest for images captured may be known. Moreparticularly, the [X, Y, Z] location (e.g., latitude, longitude, andaltitude) of an object or location seen within the images or locationseen in each image may be determined. Similarly, the altitude,orientation, roll, pitch, yaw, and the location of the LIDAR scanner 22relative to the surface 16 and/or object of interest for collection ofdata points may be known. More particularly, the [X, Y, Z] location(e.g., latitude, longitude, and altitude) of a targeted object orlocation may be determined.

The platform 12 may be piloted and/or guided through an image capturingpath that may pass over a particular area of the surface 16. In someembodiments, the image capturing path may follow one or more utilitylines. The number of times the platform 12 and/or oblique imagecapturing devices 18 a and 18 b and/or vertical image-capturing device20 pass over the area of interest may be dependent at least in part uponthe size of the area and the amount of detail desired in the capturedimages.

As the platform 12 passes over an area of interest, a number of obliqueimages and/or nadir images may be captured by the obliqueimage-capturing devices 18 a and 18 b and/or the verticalimage-capturing device 20. In some embodiments, the images may becaptured and/or acquired by the oblique image-capturing devices 18 a and18 b, and/or the vertical image-capturing device 20 at predeterminedimage capture intervals that may be dependent, at least in part, uponthe velocity of the platform 12. For example, the safe flying height fora fixed wing aircraft may be a minimum clearance of 2,000′ above thesurface 16, and may have a general forward flying speed of 120 knots. Inthis example, the oblique image-capturing devices 18 a and 18 b maycapture 1 cm to 2 cm ground sample distance imagery, and the verticalimage-capturing device 20 may be capable of capturing 2 cm to 4 cmground sample distance imagery.

The image data signals 40 a, 40 b and 42 corresponding to each imageacquired may be received by and/or stored within the one or morememories 90 of the image capturing computer system 36 via the I/O port76. Similarly, the location signals, 52, 54, 56, 58, 60, 62, and/or 64corresponding to each captured image may be received and stored withinthe one or more memories 90 of the image-capturing computer system 36via the I/O port 76. The LIDAR scanner signals 46 may be received andstored as LIDAR 3D point clouds.

Thus, the location of the oblique image capturing devices 18 a and 18 b,and/or the location of the vertical image-capturing device 20 relativeto the surface 16 at the precise moment each image is captured isrecorded within the one or more memories 90 and associated with thecorresponding captured oblique and/or nadir image.

The processor 70 may create and/or store in the one or more memories 90,one or more output image and data files. For example, the processor 70may convert image data signals 40 a, 40 b and/or 42, location signals,52, 54, 56, 58, 60, 62, and/or 64, and the LIDAR scanner signals 46 intocomputer-readable output image, data files, and LIDAR 3D point cloudfiles. The output image, data files, and LIDAR 3D point cloud files mayinclude a plurality of captured image files corresponding to capturedoblique and/or nadir images, positional data, and/or LIDAR 3D pointclouds corresponding thereto.

Output image, data files, and LIDAR 3D point cloud files may then befurther provided, displayed and/or used for obtaining measurements ofand between objects depicted within the captured images, includingmeasurements of the heights of such objects. In some embodiments, theimage capturing computer system 36 may be used to provide, displayand/or obtain measurements of and between objects depicted within thecaptured images. Alternatively, the image capturing computer system 36may deliver the output image, data files, and/or LIDAR 3D point cloudsto one or more processors, such as, for example, the processors 82illustrated in FIG. 4 for processors 82 to provide, display and/orobtain measurement.

In some embodiments, delivery of the output image, data files, and/orLIDAR 3D point cloud files may also be by physical removal of the filesfrom the image capturing computer system 36. For example, the outputimage, data files, and/or LIDAR 3D point cloud files may be stored on aremovable storage device and transported to one or more processors 82.In some embodiments, the image capturing computer system 36 may provideat least a portion of the display and/or determine at least a portion ofthe measurements further described herein.

For simplicity, the following description for measurement of objects ofinterest as described herein includes reference to utility wires,utility poles, and utility towers, however, it should be understood byone skilled in the art that the methods described herein may be appliedto any structure of interest. For example, the methods may be applied toa building structure, such as a roof, wherein the roof is the object ofinterest.

Referring to FIGS. 5 and 6, the output image file and data files may beused to geo-reference the oblique and/or nadir images. Exemplary methodsfor georeferencing the imagery may be found in at least U.S. Pat. Nos.7,424,133 and 5,247,356, which are hereby incorporated by reference intheir entirety.

The LIDAR 3D point cloud files may be processed and geo-referenced. Forexample, the LIDAR 3D point cloud files may be processed andgeo-referenced using software such as Reigl's RiProcess application,distributed by Reigl located in Horn, Austria. Generally, processing ofthe LIDAR 3D point cloud files may include classifying points in thedata into at least three categories: objects of interest 100 (e.g.,towers 114, utility wires 110), background structures 102 (e.g.,background vegetation, background structures), and surface points 16(e.g., ground points). For example, the LIDAR post processing softwaremay classify points as being the surface 16, e.g., ground, utility wires110, towers 114, and/or foliage or other background items. The towers114 can be utility towers configured to support the utility wires 110.The towers 114 can be implemented in a variety of forms, such as H-styleutility towers, utility poles, steel truss style utility towers,concrete utility towers and combinations thereof. In some embodiments,the classifications listed above may be further subdivided as needed.

Referring to FIGS. 4, 5, and 6, in some embodiments, the images and/or3D point cloud files can be scanned for horizontal objects of interestto locate utility wires 110, for example. Scanning for horizontalobjects of interest, such as the utility wires 110, may be aided by theuse of a geographical information system (GIS) data system 120illustrated in FIG. 4. For example, the GIS data system 120 may includedata from a utility company. GIS data may include, but is not limitedto, right of way centerlines, GIS data for location of towers 114, GISdata for utility wires 110, Computer Aided Design (CAD) data for theutility wires 110, and/or the like.

In some embodiments, the GIS centerline vector data may be used toautomatically follow the path of the utility network. The GIS centerlinedata is typically maintained by the utility companies and may includethe geographical position on the Earth of individual towers 114;however, such data may not be updated and/or may be changed. Thegeographical position can be in any suitable coordinate system, such asLatitude/Longitude. The centerlines, however, may remain largelyunchanged as they may typically be tied to a property boundary.

If the GIS data is inaccurate and/or unavailable, utility wires 110 mayalso be identified using either LIDAR 3D point cloud files and/or theimage data without the use of GIS data. For example, utility wires 110may generally be relatively straight lines and distinctive as comparedto other structures within the image. In three-dimensional space,utility lines 110 may be above ground and at a relatively consistentelevation range. As such, standard edge detection algorithms may be usedto identify the utility lines 110. Standard edge detection algorithmsmay include, but are not limited to, a Laplacian filter and/or the like.Additionally, in some embodiments, a Hough Transform and/or similaralgorithm, may determine the long straight feature of the utility wires110.

In some embodiments, a Gabor filter may be used to identify the utilitywires 110. The general use of a Gabor filter in identifying utilitylines is described in Mu, Chao, et al. “Power lines extraction fromaerial images based on Gabor filter.” International Symposium on SpatialAnalysis, Spatial Temporal Data Modelling, and Data Mining.International Society for Optics and Photonics, 2009, which is herebyincorporated by reference in its entirety. This method may be furthermodified to identify utility wires 110 and cross bars 112 of the towers114. Even further, the method may be modified to apply photogrammetry toautomatically isolate features in the oblique image(s) as discussed infurther detail herein.

For LIDAR 3D point cloud files, intensity values of points may beidentified and reviewed to determine the location of the utility wires110. Generally, parallel lines having periodic perpendicular edges maybe identified as utility wires 110. Additional LIDAR data points of theLIDAR 3D point cloud file may be discarded if the LIDAR data points donot correspond to the parallel lines and/or periodic perpendicularedges. For example, single lines having no close parallel line (e.g.,within 15′ or less, for example) may be discarded. Additionaldiscrimination may be performed if there are no identifiable cross arms112 in the LIDAR data points of the LIDAR 3D point cloud file. Forexample, if there are no periodic edges running perpendicular toparallel lines, the points are probably not associated with utilitywires 110.

Once utility wires 110 are identified, a wire centerline W_(C) may bedetermined to follow the utility corridor. In some embodiments, the wirecenterline W_(c) may be determined using a line fitting algorithm (e.g.,RANSAC least squares algorithm). Using the wire centerline W_(C) as aguide, measurements may be taken at predetermined increments of theutility corridor along the wire centerline W_(C). In some embodiments,the increments may be less than the height of the smallest tower 114being searched. At each increment, a search may be performed to identifyone or more clusters of LIDAR data points corresponding to one or moretowers 114, cross arms 112, and/or utility wires 110.

LIDAR data points for utilities may further be discarded based onelevation. For example, if the LIDAR data point(s) are unclassified(i.e., not classified as an object of interest 100, backgroundstructures 102, or surface 16), then the unclassified points within apredetermined distance of the lowest elevation points that areclassified may be discarded. These points may be discarded as they mayrelate to the surface 16 and/or background vegetation. Unclassifiedpoints above the lowest elevation points that are classified may beconsidered to be part of the tower 114. Typically, taller vegetation maybe kept below utility lines 110, and as such, vegetation point may notbe included in the search. In identifying vegetation in relation totowers 114, the algorithm may also look for an increased number ofpoints at a predetermined radius (e.g., 30′ radius) from a search pointhaving unclassified points, since such points will not be related toutility wires 110 if they are vegetation.

In some embodiments, towers 114, may be identified using catenary curvesof the utility lines 110. For example, as illustrated in FIG. 7, utilitywires 110 generally form parabolic curves 130 and 132 meeting at adistinct attachment point 134. Analyzing the utility wires 110 to findadjacent and intersecting parabolic curves 130 and 132 may determine thedistinct attachment point 134 at the location of intersection. Thetowers 114 location may be found at the distinct attachment point 134.

In some embodiments, once a cluster of LIDAR data points is identified,an algorithm may calculate a center of mass and grow the cluster suchthat it includes all of points reasonably within the area of interest.For example, a point density algorithm may be used to grow the clustersuch that new points may be below a selected density threshold. A ConvexHull algorithm may then be used to isolate the cluster of points andidentify a grouping of points, classifying the points as the tower 114.

Referring to FIG. 5, cross arms 112 may be identified within the obliqueand/or nadir images. Cross arms 112 may be identified as horizontallyextending, beam-like structures located close to or at relatively thesame elevation of the utility wires 110. In some embodiments, cross arms112 may have a major axis extending near perpendicular (e.g., within 10degrees of perpendicular) to and at relatively the same elevation of theutility wires 110. In some embodiments, the search and/or scanning maybe aided by the use of GIS data for the location of the towers 114and/or from the CAD data of the towers 114.

In some embodiments, the output image files and/or the LIDAR 3D pointcloud files may be scanned for horizontally extending structures (e.g.,having a major axis extending horizontally) indicative of cross arms112, as discussed above. FIGS. 8A and 8B show a LIDAR 3D point cloudwith FIG. 8B as a magnified view of the portion around the object ofinterest 100. In FIGS. 8A and 8B, the LIDAR 3D point cloud files mayclassify objects of interest 100 and background vegetation 102. Theutility wires 110 may be identified in the LIDAR 3D point cloud fileand/or the output image files. As such, the cross arms 112 may beidentified as horizontal structures near perpendicular to and/orinteresting with the utility wires 110 as illustrated in FIG. 8B. Insome embodiments, the industry standard edge detection and lineidentification algorithms may be used to determine the location of theutility wires 110 using the LIDAR data files.

Utility wires 110 may make a turn in the utility line. At such a turn,the angle of the structure of the cross arm 112 may not beperpendicular, but may typically be either perpendicular to a singleutility wire 110 or the other wire, bisecting the angle formed by thetwo utility wires, or somewhere in between the perpendiculars and theangle bisector.

Once the cross arms 112 are identified within the LIDAR 3D point cloudfiles and/or the output image files, the vertical structures beneath thecross arms 112 may be identified. Vertical structures may include towers114, and/or insulators 116. The vertical structures may be identifiedusing LIDAR data points and/or algorithms capable of isolating pointscorresponding to the vertical structures.

Prior to or after the horizontal and the vertical structures have beenidentified in the image files, the images files can be processed tocreate a pre-calculated tessellated ground plane for each of the imagesfiles. The tessellated ground plane can be implemented as a data file ordata table having elevation values that are correlated to specificgeographical locations on the surface 16 of the Earth. The tessellatedground plane includes a plurality of individual facets having respectiveelevations. Adjacent pairs of facets share at least two vertices. Eachfacet has a respective pitch and slope. Tessellated ground plane can becreated based upon various data and resources, such as, for example,topographical maps, and/or digital raster graphics, survey data, andvarious other sources. Techniques for making and using an exemplarytessellated ground plane is described in U.S. Pat. No. 7,424,133, whichis hereby incorporated herein by reference.

Referring to FIGS. 6 and 9, the tessellated ground plane can besupplemented with further information and/or data points indicative ofTGP Vertical planes Pv representative of a mathematical model of and topermit measurements to be made on an object of interest, such as avertical structure. For example, a TGP vertical plane P_(V) may beplaced transversely through the tower 114 and may be relatively parallelto the orientation of the cross arms 112. Generally, the TGP verticalplane P_(V) of each tower 114 may be formed by identifying points of thetower 114 positioned at a distance farthest from the wire centerlineW_(C) in the (x,y) direction and generally perpendicular to the utilitywires 110. The TGP vertical plane P_(v) may be formed of TGP verticalplane data of real-world three-dimensional location valuesrepresentative of at least two points on the object of interest depictedin the oblique image and positioned at a distance farthest from acenterline of the object of interest. These points may correspond to theends 116 a and 116 b of the cross arms 112. For example, in FIGS. 6 and8, points (X₁, Y₁, Z_(1A)) and (X₃, Y₃, Z_(3A)) are positioned at thefarthest extent of the tower 114 away from the centerline of the utilitywires 110 (X₂, Y₂, Z_(2A)). Connecting corresponding points at roughlythe same vertical elevation may produce a three-dimensional line roughlycorresponding to a center C of the cross arm 112. Optionally, a linefitting algorithm may be used to manipulate the line L₁ such that theline L₁ is as close to parallel to the cross arm 112 data points and the“center of mass” of the tower 114. The TGP vertical plane Pv may beformed such that it terminates the height of the cross arms 112, oranywhere on the pole and/or tower 114. For example, the TGP verticalplane Pv may be formed such that it extends to the very top height ofthe pole and/or tower 114. In this example, any and all features on thepole and/or tower 114 may be identified and/or measured using the singleray projection method once the TGP vertical plane Pv is incorporatedinto the standard ground plane.

Using these points (X₁, Y₁, Z_(1A)) and (X₃, Y₃, Z_(3A)) positioned atthe farthest extent of the tower 114, a line L₁ may be fittedtherebetween. The line L₁ may generally be through the “center of mass”of the structure points of the tower 114. The line L₁ may be extended ina z-direction to the top of the tower 114, and may also be extended in az-direction down to the surface 16 to form the TGP vertical plane P_(V).The TGP vertical plane data may include at least one real-worldthree-dimensional location value representative of a three-dimensionallocation where the object of interest over lies the Earth and having anelevation value indicative of an elevation of the terrain underneath theobject of interest. For example, in FIG. 6, the line L₁ may be extendedupwards in the z-direction to include points (X₁, Y₁, Z_(1C)) and (X₁,Y₁, Z_(3C)). The line L₁ may also be extended downwards in thez-direction to the surface 16 to include points (X₁, Y₁, Z_(1B)) and(X₃, Y₃, Z_(3B)). Modification of the line L₁ with Z values greater thanor lower than Z_(1A) and Z_(3A) may form an outer boundary of the TGPvertical plane P_(V).

Generally, a vast majority of structures on the tower 114 may lie on theTGP vertical plane P_(V). As such, the TGP vertical plane P_(V) may beused as a facet within the tessellated ground plane (TGP) for single rayprojection measurement methods as described in U.S. Pat. No. 7,424,133,which is hereby incorporated by reference in its entirety. In thisinstance, the one or more processors 82 may receive one or more signalindicative of a selection and pixel location within a displayed image ofa first point and a second point on the tower 114 depicted within thedisplayed oblique image. The one or more processors 82 may then retrievefrom a data file, location data indicative of a position and orientationof an image capturing device (e.g., the oblique image capturing devices18 a and 18 b) used to capture the displayed oblique image, and a TGPvertical plane approximating a center of mass of the tower 114. The oneor more processors 82 may then determining real-world locations of thefirst point and the second point utilizing the pixel location of the oneor more selected points within the oblique image, the location data andthe TGP vertical plane data using the single ray projection measurementmethods.

Referring to FIGS. 10, 11A and 11B, in order to compensate, at least inpart, for changes in elevation and resultant inaccuracies in themeasurement of and between objects of interest within an image, locationof points within the oblique and/or nadir image may be determined usingthe TGP vertical plane P_(V) for reference as a facet within thetessellated ground plane.

Element 150 illustrates the boundaries of a view of a metric obliqueimage. The oblique image view 150 includes a view of the tower 114 seenwithin the LIDAR data points. The TGP vertical plane P_(V) is shownextending through the tower 114. Generally, the geo-location of a pointof interest within the oblique image view 150 may be calculated bydetermining the point of intersection of a ray 152 projected from theplatform 12 towards the surface 16. For example, in some embodiments, auser may select a point in the image 150 corresponding to an object onthe tower 114. The ray 152 may be projected to intersect the TGPvertical plane P_(V) prior to the ray 152 intersecting the surface 16.For example, the ray 152 interests the vertical plane P_(V) in FIG. 10at intersection point 154. Thus, the location for the point ofintersection 154 may be determined on the tower 114 rather than a pointon the surface 16 or other tessellated ground plane.

Referring to FIG. 10, the TGP vertical plane P_(V) may also be used todetermine a length L_(O) of an object on the tower 114. When a firstpoint of the object of interest on the tower 114 is selected, the rowand column (e.g., (X, Y) location) of that pixel in the image 150 may beused to calculate projection of the ray 152 towards the surface 16. Thealgorithm may then identify where the intersection point 154 of the ray152 occurs on the TGP vertical plane P_(V) and report the intersectionpoint 154 location on the TGP vertical plane P_(V). Thethree-dimensional location of the intersection point 154 can bedetermined using bilinear interpolation using the coordinates(X₁,Y₁,Z_(1A)), (X₁,Y₁,Z_(1B)), (X₃,Y₃,Z_(3A)), (X₃,Y₃,Z_(3B)). If apixel corresponding to a second point of the object of interest on thetower 114 is selected within the image 150, the algorithm may again beused to produce a second ray 156 and identify the intersection point 158of the vertical plane P_(V). The distance between the first intersectionpoint 154 and the second intersection point 158 may be determined (e.g.,using Pythagorean Theorem), resulting in the length L_(o) of the objectmeasured in the image 150.

Generally, in using the TGP vertical plane P_(V), if an object ofinterest is located 5′ off of the TGP vertical plane P_(V) when theoblique image view 150 is captured at 2,000′ over ground at a roughly 45degree angle, an object 50′ up on the tower 114 may be over 2,750′ away.Thus, being 5′ away from the TGP vertical plane P_(V) may only result ina measurement scaling error of less than 0.2% of the actual measurement.By contrast in using a facet conforming to a portion of the surface 16,50′ below the object (i.e., surface 16), there may be a contribution of14× the amount of error due to relative path length (i.e., 50′ down and50′ over, due to a 45 degree view angle). As such, the correspondingpoint on the ground may be 70′ away (i.e., 14× the 5′ distance).Additionally, the ground plane (i.e., surface 16) may not be parallel tothe object being measured.

Referring to FIGS. 6 and 11A, the TGP vertical plane P_(V) may also beused to determine a height H above ground of an object of interest. Forexample, the TGP vertical plane P_(V) may be used to determine theheight H above ground of connection points of the utility wires 110 tothe tower 114. A user (e.g., human, processor) may select one or morepixels within the image 150 depicting the insulator 116 a, 116 b, and/or116 c. The insulators 116 a-116 c are generally the mechanism used toconnect the utility wires 110 to the tower 114. An algorithm may use apixel location (e.g., x, y) within the image 150 that is indicative ofthe user selected point to project the ray 152 through the focal planeand down towards the surface 16. Using the ray, the location of theintersection point 154 located on the insulators 116 a, 116 b, and/or116 c, may be determined on the TGP vertical plane P_(V). The secondpoint of intersection may be selected on the surface 16 providing thepoint of intersection on a facet within the tessellated ground plane158. The Z distance between the two points 154 and 158 in space may bedetermined to be the height H above ground for the connection point.

It should be noted that the tessellated ground plane 158 having facetsconforming to the contours of the surface 16 of the Earth, as describedin U.S. Pat. No. 7,424,133, may also be determined using data pointscollected by the LIDAR scanner 22. Using the normal tessellated groundplane 158, the intersection of the ground may be determined as theintersection of the TGP vertical plane P_(V) with the tessellated groundplane 158. Using the tessellated ground plane 158, the measurement ofthe height H may be increased in accuracy in some embodiments, and alsomay be used for purposes of thermal line ratings.

Referring to FIGS. 6, 11A and 11B, to further increase accuracy, a usermay select the same or similar connection point on the tower 114 in twoor more oblique image views 150 and 150 b. For example, the user mayselect the insulator 116 a on the tower 114 in a forward facing obliqueimage 150 as illustrated in FIG. 11A and a rear facing oblique image 150b as illustrated in FIG. 11B. In each oblique image view 150 and 150 b,the heights H and H₂ respectively of the insulator 116 a may bedetermined as described herein. The intersection points 154, 154 b and156, 156 b may be found using standard stereo pair photogrammetrytechniques such that the location of each point may be determined withincreased accuracy as compared to using a single image. The tessellatedground plane 158 may also be used to determine the heights H and H₂increasing accuracy of the determination even further. The tessellatedground plane 158 may further increase accuracy due to the errorbalancing nature of stereo-photogrammetry, however, single rayprojection measurements may also be used to review the measurements forproper selection.

In some embodiments, the stereo analysis using standard stereo pairphotogrammetry techniques may be automated or substantially automated.Generally, a corner detection algorithm may be used to find points ofinterest in two separate oblique image views 150 and 150 b for anobject. A correlation for the two points of interest may be determinedto identify common points between the two points of interest. Thestrongest correlation may generally be on the desired object.

Using this example, by selecting a pixel indicative of a connectionpoint (e.g., insulator 116 a) in a first oblique image view 150, the ray152 may be determined. The resulting intersection point 154 may be usedto select a second oblique image view 150 b from an opposing direction.The TGP vertical plane P_(V) may then be used to find an end of theinsulator 116 a. A standard corner detection algorithm and/or acorrelation algorithm may then be used to find a pixel indicative of theend of the insulator 116 a in the second image 150 b. Once the end ofthe insulators 116 a in the second image 150 b is located, the locationof the pixel within the second image 150 b, the TGP vertical planeP_(V), and the camera position and orientation of the second image 150 bmay be used to cast a second ray 152 b through the end of the insulator116 a in the second image 150 b. The resulting two rays 152 and 152 bmay then be used in industry standard stereo photogrammetry to locatethe intersection points 154, 154 b and 156, 156 b. The resultingidentification and measurement of and between the points 154, 154 b and156, 156 b may be further incorporated into CAD modeling, thermal lineratings, engineering plans, and/or any other use for three-dimensionalpoint determinations. Even further, identification and/or measurementbetween multiple points between multiple images may aid in formation ofMethod 1 structure models as known in the industry.

Referring to FIGS. 12-17, identification of matching points between twoopposing oblique images 150 a and 150 b may also be identified using aGabor filter. The orientation and spatial frequency of the Gabor filtermay be tuned such that the filter acts as an oriented bandpass filter.

Referring to FIGS. 12 and 13A, in a nadir image 200, the utility wires110 include distinct oriented spatial frequencies that may be identifiedusing a Gabor filter providing a Gabor filters image 13A. Theorientation of the utility wires 110 in the nadir image 200 may beidentified based on the orientation of the platform 12 (illustrated inFIG. 1) during flight. Additionally, identification of the spacingbetween each utility wire 110 may aid in tuning the Gabor filter toproduce maximum response, however, a reasonable estimation of thefrequency may be used.

Referring to FIGS. 13A and 13B, a maximum value threshold may isolatethe highest response from the Gabor filtered image 210 creating athreshold image 220. Once the threshold image 220 is created, linearfeatures may be identified within the threshold image 220 producingdetected utility wires 222. For example, detected utility wires 222 maybe identified within the threshold image 220 using the Hough Transform,filtering for lines that may only be at two specified orientations ofthe Gabor filter.

Referring to FIGS. 12, 14A and 14B, in some embodiments, the cross arm112 connecting the utility wires 110 may be extracted using a Gaborfilter. For example, the cross arm 112 may be extracted by rotating theorientation of the Gabor filter within a range from 85-95 degrees(preferably 90 degrees) producing a Gabor filter image 230 in FIG. 14Ahaving Gabor detected features 232. A maximum value threshold mayisolate the highest response from the Gabor filtered image 230 creatinga threshold image 240. Once the threshold image 240 is created, linearfeatures may be identified within the image similar to threshold image220 producing detected cross bar lines 232.

Since many lines within the threshold image 240 may not be continuous,the entire cross bar 112 of FIG. 12 may not be detected. As such, eachdetected utility wire 222 of threshold image 220 (illustrated in FIG.13B) may be intersected with the detected cross bar lines 242 ofthreshold image 240 (illustrated in FIG. 14B) to define endpoints 250and 252 of the cross arm 112 between the utility wires 110 asillustrated in FIG. 15A. An extension 254 may be applied to the detectedcross bar 242 based on the defined maximum and minimum endpoints 250 and252 as illustrated in FIG. 15B.

Referring to FIG. 16, a correlation of endpoints 250 and 252 for thedetected cross bar 242 may be initiated using an oblique image 260 ofthe object identified in the nadir image of FIG. 12. The correlationregion in the oblique image 260 may be produced by projecting theendpoints 250 and 252 of the detected cross bar 242 into the obliqueimage 260, and correlating a region around epipolar lines 262 a and 262b for each endpoint 250 and 252 of the detected cross bar 242. Forexample, the detected cross bar 242 may be correlated such that thedetected cross bar 242 substantially lies on the cross bar 212.

Additionally, matching points between opposing oblique images havingdetected cross arms 242 may be identified. Using these points, a regionof interest may be determined around each detected cross arm 242. Otherfeatures of the tower 114 may then be further identified using theregion of interest. In one example, as illustrated in FIG. 17 a secondTGP vertical plane P_(V2) may be defined and/or extended from theidentified cross arm 112 such that the TGP vertical plane P_(V2) extendsa pre-determined distance from the identified cross arm 112 and containsremaining features to be identified using methods as described herein.

Referring to FIGS. 18A-18D, in some embodiments, a template may be usedto determine location of objects of interest on structures (e.g., crossbars 112 on tower 114). For example, FIGS. 12A-12D illustrate anexemplary embodiment of a utility template 159 for use in determininglocation of objects of interest, such as towers 114. Generally, a usermay be supplied with one or more template structures. The templates maycorrelate with identified structures within the oblique images and/orvertical images. For example, the template 159 illustrated in FIGS.12A-12D is a template of a “H” style tower 114. The user (e.g., human,processor) may align the template 159 to the object of interest.

For example, as illustrated in FIG. 18A, the user may align a first legsection 160 a of the “H” with a first leg 113 a of the tower 114. Theuser may then laterally stretch the template 159 such that a second legsection 113 b of the “H” of the template 159 may be aligned with asecond leg 113 b of the tower 114 as illustrated in FIG. 12B. The usermay vertically stretch the template 159 such that a cross line 162 ofthe template 159 may be aligned with the cross arm 112 of the tower 114as illustrated in FIG. 12C. Finally, the user may adjust one or morelengths of the cross line 162 of the template 159 to substantially lieon the insulators 116 a-116 c of the tower 114. The template 159, onceadjusted, may lie directly on the tower 114. By determining the verticalplane P_(V), as described herein, the location of the tower 114 in spacemay be known. The same template 159, as such, may be projected onto oneor more images with opposing views of the tower 114. Slight adjustmentsmay be made to compensate for any minor errors in the position and/ororientation measurements (e.g., position and/or orientation measurementsdue to camera error).

Standard stereo triangulation may also be used to determine location ofeach end of the line segments within the template 159. With thedetermination of the location of each end of the line segments withinthe template 159, the structure and location of the tower 114 withinspace may be determined and applied to one or more additional obliqueand/or nadir images.

Referring to FIGS. 2, 6 and 9, the TGP vertical plane P_(V) may also aidin generation of additional three-dimensional points to augment thethree-dimensional point cloud generated by the LIDAR scanner 22. TheLIDAR scanner 22 may not identify a significant number of points onfaces of the towers 114, as will be explained below. If, the LIDARscanner is positioned and aimed towards nadir, upper structures of thetower 114 may obscure elements below the tower 114. In addition, trulyvertical structures may not produce a significant return to the LIDARscanner 22 if the LIDAR scanner 22 is aimed towards nadir. If the LIDARscanner 22 is tilted forward or backward at an angle to try and producea greater return, the point density may still be low due to the cosineeffect. For example, if the LIDAR scanner 22 is tilted forward by 10degrees, and the LIDAR scanner collects data at 50 points per squaremeter such that there is a 6″ sampling distance on the surface 16.Because the laser beam of the LIDAR scanner 22 intersects the tower 114at an angle of 10 degrees, a 100 foot tall pole may only appear to beabout 17 feet in length, and as such, may only get approximately 34points over the height of the tower 114 (as compared to over 200 pointsproduced the same distance on the surface 16). Additionally, the angleof incidence of the imagery may be closer to 45 degrees. The same 6″resolution may produce 140 points on a face of the tower 114. Theimagery, however, isn't at the same 6″ resolution, it may be at a 2″resolution. This may produce increased resolution in each direction,such that if each pixel yields a correlation point, more than 420 pointsalong the height of the tower 114 may be produced. Even further, thepixel resolution being generally smaller than the diameter of the tower114, multiple points across the diameter of the tower 114 may beproduced.

Referring to FIGS. 2 and 19, additional three dimensional points toinclude in the 3D point cloud may also be determined using successiveoblique image (e.g., a first oblique image view 150 c and a secondoblique image view 150 d). Both the first oblique image view 150 c andthe second oblique image view 150 d may include an object of interest,such as, for example, the tower 114. The TGP vertical plane P_(V) may bedetermined using the methods as described herein. Once identified, theTGP vertical plane P_(V) may be used to select one or more overlappingoblique images oriented in the same direction (e.g., both forwardlooking camera orientations or both rearward looking cameraorientation). The TGP vertical plane P_(V) may be used to identify thelocation of the tower 114 in each image 150 c and 150 d as described indetail herein using either one or both of rays 152 c and 152 d. Thelocations of the tower 114 in each image 150 c and 150 d may be used inan automated point matching algorithm (e.g., Semi Global Image Mappingalgorithm) to find corresponding points between the two images 150 c and150 d.

Either one of the projected rays 152 c and 152 d may then be used in asingle ray-projection algorithm or (both of the rays 152 c and 152 d ina standard stereo photogrammetry algorithm) to find the real-world,three-dimensional location of the point of intersection that may beadded to the point cloud produced by the LIDAR scanner 22. It should benoted that there may be alignment errors (e.g., inertial navigationsystem (INS) errors), and as such, the point cloud may be misalignedwith the results produced by the LIDAR scanner 22. These two pointclouds may be related through a similarity transform with uniform scale.The transform (e.g., iterative closest point algorithm) may iterativelyestimate the distance between the results produced by the LIDAR scanner22 and a point cloud produced by the images 150 c and 150 d. Theresulting point cloud from combining results produced by the LIDARscanner 22 and the point cloud produced by the images 150 c and 150 dmay be denser and include points located on multiple faces ofstructures. For example, having two oblique image capturing devices 18 aand 18 b as illustrated in FIG. 1 may produce images on multiple sidesof structures as compared to having only a single LIDAR scanner titledin a single direction gathering points on a single side of a structure.FIG. 20 is a three-dimensional point cloud generated from stereo pairoblique images showing points of the ground 16 and points of utilitytowers 114. As shown, the point cloud produced by the oblique imagesincludes points on the horizontal surfaces (e.g., ground 16) and pointson the vertical surfaces (e.g., vertical facet of the utility towers114).

Although the preceding description has been described herein withreference to particular means, materials and embodiments, it is notintended to be limited to the particulars disclosed herein; rather, itextends to all functionally equivalent structures, methods and uses,such as are within the scope of the appended claims.

What is claimed is:
 1. An automated method, comprising: capturing imagesand three-dimensional LIDAR data of a geographic area with an imagecapturing device and a LIDAR system, the images depicting an object ofinterest and the three-dimensional LIDAR data including the object ofinterest, the image capturing device capturing the images of a verticalsurface of the object of interest at one or more oblique angle, and theLIDAR system capturing the three-dimensional LIDAR data of a horizontalsurface of the object of interest at a nadir angle; analyzing the imageswith a computer system to determine three dimensional locations ofpoints on the object of interest; and updating the three-dimensionalLIDAR data with the three dimensional locations of points on the objectof interest determined by analyzing the images to create a 3D pointcloud having a resolution greater than a resolution of thethree-dimensional LIDAR data.
 2. The automated method of claim 1,wherein the image capturing device and the LIDAR system are mounted to aplatform, and wherein capturing images and three-dimensional LIDAR dataare defined further as flying the platform over the geographic area. 3.The automated method of claim 2, wherein the platform is unmanned. 4.The automated method of claim 2, wherein the platform is manned.
 5. Theautomated method of claim 1, wherein the image capturing device and theLIDAR system are mounted to an airplane, and wherein capturing imagesand three-dimensional LIDAR data are defined further as flying over thegeographic area with the airplane.
 6. The automated method of claim 1,wherein the object of interest includes a utility tower, and wherein thestep of analyzing the images is defined further as utilizing GIS data ofa utility network to assist in locating the object of interest withinone or more images.
 7. The automated method of claim 1, wherein theobject of interest includes a utility tower, and wherein the step ofanalyzing the images is defined further as scanning the images with anedge detection algorithm to locate utility wires depicted within theimages, prior to determining three-dimensional location of points on theutility tower.
 8. The automated method of claim 1, further comprisingobtaining measurements of the object of interest.
 9. The automatedmethod of claim 1, further comprising obtaining measurements between theobject of interest and one or more other objects captured in the 3Dpoint cloud.
 10. The automated method of claim 1, further comprisingforming a model of the object of interest from the 3D point cloud.
 11. Asystem, comprising: an image capturing device capturing images of ageographic area, the images depicting at least a vertical surface of anobject of interest at one or more oblique angle; one or more LIDARsystem capturing at a nadir angle three-dimensional LIDAR data of thegeographic area such that the three-dimensional LIDAR data includes atleast a horizontal surface of the object of interest; and a computersystem analyzing the images to determine three dimensional locations ofpoints on the object of interest and updating the three-dimensionalLIDAR data with the three dimensional locations of points on the objectof interest determined by analyzing the images to create a 3D pointcloud having a resolution greater than a resolution of thethree-dimensional LIDAR data.
 12. The system of claim 11, furthercomprising a platform upon which the one or more image capturing devicesand the one or more LIDAR system are mounted.
 13. The system of claim12, wherein the platform is unmanned.
 14. The system of claim 12,wherein the platform is manned.
 15. The system of claim 11, furthercomprising an airplane upon which the one or more image capturingdevices and the one or more LIDAR system are mounted such that theimages and three-dimensional LIDAR data are captured by flying over thegeographic area with the airplane.
 16. The system of claim 11, whereinthe object of interest includes a utility tower, and wherein analyzingthe images is defined further as utilizing GIS data of a utility networkto assist in locating the object of interest within one or more images.17. The system of claim 11, wherein the object of interest includes autility tower, and wherein analyzing the images is defined further asscanning the images with an edge detection algorithm to locate utilitywires depicted within the images, prior to determining three dimensionallocation of points on the utility tower.
 18. The system of claim 11,wherein the computer system further obtains measurements of the objectof interest.
 19. The system of claim 11, wherein the computer systemfurther obtains measurements between the object of interest and one ormore other objects captured in the 3D point cloud.
 20. The system ofclaim 11, wherein the computer system further forms a model of theobject of interest from the 3D point cloud.